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ABSTRACT 



As part of a study of school effectiveness, a procedure was 
developed to select schools for study based on extreme high or extreme low 
degrees of successfulness in reading. Data for all 775 public elementary 
schools in Maryland that were active in 1995 were used. Five triads of 
schools were selected, each triad consisting of two schools that were high 
and one school that was low in successfulness. The triads were: (1) high 

income suburban; (2) moderate income suburban; (3) low income urban; (4) low 
income suburban; and (5) low income rural. Achievement in schools was 
measured by the Maryland School Performance Assessment Program. As a first 
step in selection, regression models were developed to predict reading school 
means. Then the regression models were used to generate selection indices. 
Schools were then grouped, first into income levels, and then into levels of 
urbanicity. The 5 groups from which to select a triad each were then found to 
be 39 low income urban, 55 low income suburban, 11 low income rural, 28 
moderate income suburban, and 78 high income suburban. In each group, schools 
were ordered on the basis of the selection index. The two highest -scoring and 
lowest-scoring schools were invited. If a school did not agree to 
participate, the next -higher or next-lower scoring school was invited until 
the triad was selected. One appendix, which contains seven tables, discusses 
the school -level successfulness indices across the state achievement testing 
program scores, and the other compares test score residuals with the school 
performance index and its residuals. (Contains six references.) (SLD) 
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Selection of Higher Successfulness and Lower Successfulness Schoolsl 

William D. Schafer 

University of Maryland, College Park 

The purpose of this phase of our research was to develop and implement a 
procedure to select schools based on extreme high or extreme low degrees of 
successfulness in reading. Five triads of schools were selected, each triad 
consisting of two schools that were high and one that was low on 
successfulness. The five triads were: (1) high income suburban, (2) moderate 

income suburban, (3) low income urban, (3) low income suburban, and (5) low 
income rural. The data we had available to make these selections was limited 
to the school-level measures and indicators provided to us by the Maryland 
State Department of Education. 

Schools and Variables 



Achievement in schools was measured by the Maryland School Performance 
Assessment Program (MSPAP) . Six content areas are tested in grades 3 and 5 by 
MSPAP : reading, writing, language arts, mathematics, science, and social 

studies. Testing typically takes place over the span of a week and involves 
group and individual activities used to measure applications of knowledge and 
skills, but the way the domain is organized differs across content areas. 
Different students complete different activities that make up the scoring 
events by which the school is assessed. Individual scores on the six content 
area scales are estimated using item response theory models. The content area 
scales are equated across years so that comparisons over time are meaningful. 
For a more detailed overview, see Yen and Ferrara (1997). 



Data for all 775 Maryland public elementary schools that were active in 
1995 were forwarded to us by the Maryland State Department of Education. The 
four academic years 1992 through 1995 were included. For each year, variables 
were included that represented: 



school district 
urbanicity 
enrollment 
attendance 
mobility 
absenteeism 
percent Title I 
percent ESOL v 

percent special education 
percent free or reduced price meals 
gender distribution 
ethnic distribution 
number of students in grades 3 
numbers and percents in grades 
percent satisfactory in grades 
scale score mean and st. dev. 
test 
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Differentiation of Schools by Successfulness 



Differentiation of more and less successful schools has been attempted 
in the past but with mixed results. Rowan, Bossert, and Dwyer (1983) 
discussed four basic ways to approach this problem. These are: (1) absolute 

instructional outcome measures such as proportion below grade level, (2) 
evaluation of trends in grade levels across years, (3) evaluation of trends in 
cohorts across years such as increases relative to national norms, and (4) 
residuals from predictions using demographic composition. However, Mandeville 
and Anderson (1987) characterize approaches in which achievement is regressed 
onto both socioeconomic status (a component of demographic composition) and 
prior achievement (Dyer, Linn, and Patton, 1969) as having the most empirical 
support . 

Controlling for prior-year achievement of the same students and 
socioeconomic status indicators, Mandeville and Anderson (1987) found the 
overall predictability (squared multiple correlations) of mathematics to be in 
the range of .34 to .46 and of reading to be in the range of .48 to .76 across 
grades one to four using school-level data for over 500 South Carolina 
schools. They then used these equations to find the residuals of the schools 
and standardized them by their estimated standard errors. The correlations 
between mathematics and reading residuals ranged from .60 to .70 across the 
four grades. However, the median cross-grade correlations were only .06 for 
mathematics and .13 for reading. 

Mandeville (1988) further analyzed these data along with the following 
year’s data on the same schools. He evaluated the consistency of the 
standardized residuals. The correlation between the two years (different 
students) ranged from .34 to .60 in mathematics and from .36 to .65 in reading 
across the four grades; a composite sum of the standardized residuals 
correlated .46 for mathematics and .41 for reading and was judged not to 
improve stability. The eight correlations between reading and math that held 
year in common (same students) ranged from .59 to .74. There were six 
correlations between pairs of the four grades each year, yielding 12 cross- 
grade correlations for each subject matter area. These cross-grade 
correlations ranged from .00 to .19 in mathematics and from -.02 to .18 in 
reading. In reviewing these two sets of data, Mandeville (1988) suggested 
that teacher cohorts instead of schools should be the focus of studies of 
successfulness • 

The need for student-based data to be included in a regression-based 
procedure for differentiation of schools on successfulness was considered by 
Mandeville (1988), who concluded that his basic findings would change little. 
However, Webster, Mendro, Bembry, and Orsak (1995) described a study that 
compared ranking procedures using student-based data with school-level data 
only and found different rankings. Webster, Mendro, Bembry, and Orsak (1995) 
also calculated ranks according to several different algorithms using student- 
level data and found them to be almost interchangeable. 

Method 



Based on available data and informed by the studies reviewed, our 
selection of the five triads of schools was accomplished in several steps. 
Appendices A and B explore these steps for content areas other than reading. 
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1. We developed regression models to predict Reading school means. 

PROCESS: Using weighted regression, weighting by the inverse variance 

of error of the mean, we predicted 199X mean MSPAP reading score at each 
school at each grade level (third and fifth) using as predictors: 

elementary enrollment 
elementary attendance rate 
percent entrants 
percent withdrawls 

percent absent less than five days 
percent absent more than twenty days 
percent special education 
percent free or reduced price meals 
percent Indian (American or Alaskan Native) 
percent Asian or Pacific Islander 
percent African American 
percent Hispanic 

percent accounted for on the 199X MSPAP reading assessment 

\ 

We then repeated the regressions adding 199 (X-l) mean MSPAP reading 
score at that grade level. This was done in order to have a measure that 
could tap gains in reading performance along with the previous measure that 
evaluates absolute performance in relation to the demographic variables. 

RESULT: Four regression models for each of three years were developed 

(third & fifth with & without prior mean reading score), 1995, 1994, 1993. 

EVALUATION: The multiple R-Square values for each of the twelve 

regressions were: 



1995 


Grade 


Three, 


Without 


Prior 


Reading 


Score: 


.73 


1995 


Grade 


Three, 


With 


Prior 


Reading 


Score: 


.80 


1995 


Grade 


Five, 


Without 


Prior 


Reading 


Score: 


.73 


1995 


Grade 


Five, 


With 


Prior 


Reading 


Score: 


.79 


1994 


Grade 


Three, 


Without 


Prior 


Reading 


Score: 


.76 


1994 


Grade 


Three, 


With 


Prior 


Reading 


Score: 


.82 


1994 


Grade 


Five, 


Without 


Prior 


Reading 


Score: 


.74 


1994 


Grade 


Five, 


With 


Prior 


Reading 


Score : 


.80 


1993 


Grade 


Three, 


Without 


Prior 


Reading 


Score : 


.78 


1993 


Grade 


Three, 


With 


Prior 


Reading 


Score: 


.84 


1993 


Grade 


Five, 


Without 


Prior 


Reading 


Score: 


.71 


1993 


Grade 


Five, 


With 


Prior 


Reading 


Score: 


.79 



The values for grade three with prior reading range from .80 to .84 and 
are most directly comparable to Mandeville's (1988) squared multiple 
correlations of .64 to .65 for grade three with percent free or reduced lunch 
and prior year reading score as predictors. The greater predictability we 
found may be due to the increased number of predictors we used and/or to our 
use of weighted regression. On the other hand, Mandeville's (1988) use of 
prior test score from the previous year on the same students should lead to a 
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higher correlation since our data on prior year were for an independent group 
of students. 

2. We used the regression models to generate selection indices. 

PROCESS: We calculated "studentized” residuals from each of the twelve 

regression models. (Studentized residuals are standardized by the estimates 
of their individual standard errors.) We then created a selection index at 
each of the three years by combining the four studentized residuals for grades 
three and five. They were unweighted in the sum since we wanted to reflect 
absolute success levels and change in level of success about equally in 
selecting schools. Then, we computed a final selection index by summing the 
selection indices, weighting each subsequent year twice the previous year. 
Although Mandeville's (1988) results suggested that we would not improve 
stability of this index very much if at all, we nevertheless wanted to reflect 
consistency in our selection index, but at the same time emphasize more recent 
data . 



In the table below, the raw residuals are labeled: 



1995 


Grade 


Three, 


Without 


Prior 


Reading 


Score: 


RDABL395 


1995 


Grade 


Three, 


With 


Prior 


Reading 


Score: 


RDCHA395 


1995 


Grade 


Five, 


Without 


Prior 


Reading 


Score: 


RDABL595 


1995 


Grade 


Five, 


With 


Prior 


Reading 


Score: 


RDCHA595 


1995 


Grade 


Three, 


Without 


Prior 


Reading 


Score : 


RDABL394 


1995 


Grade 


Three, 


With 


Prior 


Reading 


Score : 


RDCHA394 


1995 


Grade 


Five, 


Without 


Prior 


Reading 


Score: 


RDABL594 


1995 


Grade 


Five, 


With 


Prior 


Reading 


Score: 


RDCHA594 


1995 


Grade 


Three, 


Without 


Prior 


Reading 


Score: 


RDABL393 


1995 


Grade 


Three, 


With 


Prior 


Reading 


Score: 


RDCHA393 


1995 


Grade 


Five, 


Without 


Prior 


Reading 


Score: 


RDABL593 


1995 


Grade 


Five, 


With 


Prior 


Reading 


Score: 


RDCHA593 



and the selection indices are: 

RDABL395 + RDCHA395 + RDABL595 + RDCHA595 = SELECT95 

RDABL394 + RDCHA394 + RDABL594 + RDCHA594 = SELECT94 

RDABL393 + RDCHA393 + RDABL593 + RDCHA593 = SELECT93 

and the final selection index is: 

SELECT93 + 2*SELECT94 + 4*SELECT95 = SELECT 

RESULT: This selection index was used to choose outlier schools based 

on MS PAP reading performance. A high positive number indicated a high-scoring 
school and a low negative number indicated a low-scoring school. The 
selection index was scaled by dividing by a constant 7 for ease of 
interpretation. 
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EVALUATION: Following are means, standard deviations, and correlations 

among all these variables. 



Variable 


Cases 


Mean 


Std Dev 


RDABL393 


688 


-.1434 


1.4306 


RDCHA393 


678 


-.1470 


1.7353 


RDABL593 


645 


-.1678 


1.4950 


RDCHA593 


638 


-.1190 


1.5415 


RDABL394 


660 


-.1343 


1.2396 


RDCHA394 


649 


-.1009 


1.4805 


RDABL594 


651 


-.1040 


1.6332 


RDCHA594 


641 


-.0559 


1.9201 


RDABL395 


643 


-.0048 


1.4505 


RDCHA395 


676 


-.0487 


1.3030 


RDABL595 


642 


-.0609 


1.4576 


RDCHA595 


643 


-.1181 


1.4024 


SELECT93 


632 


-.4578 


3.6768 


SELECT94 


619 


-.4672 


4.1204 


SELECT95 


625 


-.1198 


3.6438 


SELECT 


551 


-.2438 


2.5849 
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RDABL393 



RDABL394 



RDCHA394 



- - Correlation Coefficients - - 
RDCHA393 RDABL593 RDCHA593 



RDABL393 


1 


.0000 




.8391 




.4344 




.2308 




.4456 




( 


688) 


( 


678) 


( 


641) 


( 


634) 


( 


632) 




P = 


♦ 


P= 


.000 


P= 


.000 


p= 


.000 


P= 


.000 


RDCHA393 




.8391 


1 


.0000 




.3431 




.2081 




.2815 




( 


678) 


( 


678) 


( 


639) 


( 


632) 


( 


628) 




P = 


.000 


P= 


♦ 


P= 


.000 


p= 


.000 


P= 


.000 


RDABL593 




.4344 




.3431 


1 


.0000 




.8646 




.2427 




( 


641) 


( 


639) 


< 


645) 


( 


638) 


( 


605) 




P = 


.000 


P= 


.000 


P= 


. 


p= 


.000 


P= 


.000 


RDCHA593 




.2308 




.2081 




.8646 


1 


.0000 




.1471 




( 


634) 


( 


632) 


( 


638) 


( 


638) 


( 


600) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


• 


P= 


.000 


RDABL394 




.4456 




.2815 




.2427 




.1471 


1 


.0000 




( 


632) 


( 


628) 


( 


605) 


( 


600) 


( 


660) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


.000 


P= 


• 


RDCHA394 




.0434 


- 


.0479 




.0928 




.0985 




.7987 




( 


622) 


( 


618) 


( 


596) 


( 


592) 


( 


649) 




P = 


.280 


P= 


.235 


P= 


.024 


p= 


.017 


P= 


.000 


RDABL594 




.2413 




.1362 




.2916 




.1758 




.3923 




( 


623) 


( 


617) 


( 


613) 


( 


608) 


( 


632) 




P = 


.000 


P= 


.001 


P= 


.000 


p= 


.000 


P= 


.000 


RDCHA594 




.0877 


- 


.0717 


- 


.1476 


- 


.2089 




.1965 




( 


614) 


( 


609) 


( 


606) 


( 


601) 


( 


628) 




P = 


.030 


P= 


.077 


P= 


.000 


p= 


.000 


P= 


.000 


RDABL395 




.3049 




.1640 




.1908 




.1053 




.4252 




( 


618) 


( 


614) 


( 


610) 


( 


606) 


( 


606) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


.009 


P= 


.000 


RDCHA395 




.1127 




.0922 




.0589 




.0286 


- 


.0919 




( 


647) 


( 


641) 


( 


615) 


( 


612) 


( 


630) 




P = 


.004 


P= 


.019 


P= 


.145 


p= 


.479 


P= 


.021 


RDABL595 




.2845 




.1896 




.3098 




.1805 




.1622 




( 


615) 


( 


611) 


( 


608) 


( 


602) 


( 


605) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


.000 


P= 


.000 


RDCHA595 




.2414 




.2086 




.2160 




.0856 




.0815 




( 


615) 


( 


611) 


( 


609) 


( 


603) 


( 


605) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


.036 


P= 


.045 


SELECT93 




.7669 




.7187 




.8529 




.7742 




.3609 




( 


632) 


( 


632) 


( 


632) 


( 


632) 


( 


597) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


.000 


P= 


.000 


SELECT94 




.2489 




.0243 




.1263 




.0388 




.7496 




( 


595) 


( 


592) 


( 


590) 


( 


586) 


( 


619) 




P = 


.000 


P= 


.555 


P= 


.002 


p= 


.348 


P= 


.000 


SELECT95 




.3273 




.2027 




.2947 




.1735 




.2599 




( 


600) 


( 


596) 


( 


592) 


( 


589) 


( 


590) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


.000 


P= 


.000 


SELECT 




.5136 




.3469 




.4950 




.3309 




.5477 




( 


551) 


( 


551) 


( 


551) 


( 


551) 


( 


551) 




P = 


.000 


P= 


.000 


P= 


.000 


p= 


.000 


P= 


.000 
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.0434 
( 622) 
P= .280 
-.0479 
( 618) 
P= .235 
.0928 
( 596) 

P= .024 
.0985 
( 592 ) 

P= .017 
.7987 
( 649) 

P= .000 
1.0000 
( 649) 

P= . 
.2168 
( 623) 

P= .000 
.1333 
( 619) 

P= .001 
.2500 
( 598) 

P= .000 
-.1576 
( 623) 

P= .000 
.0536 
( 598) 

P= .190 
.0108 
( 598) 

P= .792 
.0699 
( 589) 

P= .090 
.6915 
( 619) 

P= .000 
.0825 
( 583) 

P= .047 
.3259 
( 551) 

P= .000 
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RDABL393 

RDCHA393 

RDABL593 

RDCHA593 

RDABL394 

RDCHA394 

RDABL594 

RDCHA594 

RDABL395 

RDCHA395 

RDABL595 

RDCHA595 

SELECT93 

SELECT94 

SELECT95 

SELECT 







RDABL594 

.2413 
( 623) 

P= .000 
.1362 
( 617) 

P= .001 
.2916 
( 613) 

P= .000 
.1758 
( 608) 

P= .000 
.3923 
( 632) 

P= .000 
.2168 
( 623) 

P= .000 

1.0000 

( 651) 

P= . 

. .7826 

( 641) 

P= .000 
.2904 
( 613) 

P= .000 
-.0308 
( 622) 

P= .443 
.3603 
( 613) 

P= .000 
.0688 
( 612) 

P= .089 
.2661 
( 604) 

P= .000 
.7981 
( 619) 

P= .000 
.2910 
( 598) 

P= .000 
.5833 
( 551) 

P= .000 



RDCHA594 

.0877 
( 614) 

P= .030 
-.0717 
( 609) 

P= .077 
-.1476 
( 606) 

P= .000 
-.2089 
( 601) 

P= .000 
.1965 
( 628) 

P= .000 
.1333 
( 619) 

P= .001 
.7826 
( 641) 

P= .000 

1.0000 

( 641) 

P= . 

.2997 
( 607) 

P= .000 
.1496 
( 614) 

P= .000 
.1194 
( 607) 

P= .003 
-.0363 
( 606) 

P= .372 
-.1325 
( 597) 

P= .001 
.7464 
( 619) 

P= .000 
.1447 
( 593) 

P= .000 
.4097 
( 551) 

P= .000 



RDABL395 

.3049 
( 618) 

P= .000 
.1640 
( 614) 

P= .000 
.1908 
( 610) 

P= .000 
.1053 
( 606) 

P= .009 
.4252 
( 606) 

P= .000 
.2500 
( 598) 

P= .000 
.2904 
( 613) 

P= .000 
.2997 
( 607) 

P= .000 

1.0000 

( 643) 

P= . 

.8034 
( 638) 

P= .000 
.3477 
( 633) 

P= .000 
.2813 
( 630) 

P= .000 
.2750 
( 603) 

P= .000 
.4293 
( 591) 

P= .000 
.7796 
( 625) 

P= .000 
.8205 
( 551) 

P= .000 



RDCHA395 

.1127 
( 647) 

P= .004 
.0922 
( 641) 

P= .019 
.0589 
( 615) 

P= .145 
.0286 
( 612) 

P= .479 
-.0919 
( 630) 

P= .021 
-.1576 
( 623) 

P= .000 
-.0308 
( 622) 

P= .443 
.1496 
( 614) 

P= .000 
.8034 
( 638) 

P= .000 

1.0000 

( 676) 

P= . 

.2481 
( 632) 

P= .000 
.2741 
( 633) 

P= .000 
.1197 
( 607) 

P= .003 
.0499 
( 596) 

P= .224 
.7625 
( 625) 

P= .000 
.6287 
( 551) 

P= .000 



RDABL595 

.2845 
( 615) 

P= .000 
.1896 
( 611) 

P= .000 
.3098 
( 608) 

P= .000 
.1805 
( 602) 

P= .000 
.1622 
( 605) 

P= .000 
.0536 
( 598) 

P= .190 
.3603 
( 613) 

P= .000 
.1194 
( 607) 

P= .003 
.3477 
( 633) 

P= .000 
.2481 
( 632) 

P= .000 

1.0000 

( 642) 

P= . 

.8722 
( 638) 

P= .000 
.2875 
( 598) 

P= .000 
.2366 
( 591) 

P= .000 
.8073 
( 625) 

P= .000 
.8374 
( 551) 

P= .000 



RDCHA595 

.2414 
( 615) 

P= .000 
.2086 
( 611) 

P= .000 
.2160 
( 609) 

P= .000 
.0856 
( 603) 

P= .036 
.0815 
( 605) 

P= .045 
.0108 
( 598) 

P= .792 
.0688 
( 612) 

P= .089 
-.0363 
( 606) 

P= .372 
.2813 
( 630) 

P= .000 
.2741 
( 633) 

P= .000 
.8722 
( 638) 

P= .000 

1.0000 

( 643) 

P= . 

.2067 
( 598) 

P= .000 
.0528 
( 590) 

P= .200 
.7677 
( 625) 

P= .000 
.6442 
( 551) 

P= .000 
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SELECT93 


SELECT94 


SELECT95 


SELECT 


RDABL393 


.7669 




.2489 




.3273 




.5136 




( 632) 


( 


595) 


( 


600) 


( 


551) 




P= .000 


P= 


.000 


P = 


.000 


P= 


.000 


RDCHA393 . 


.7187 




.0243 




.2027 




.3469 




( 632) 


( 


592) 


( 


596) 


( 


551) 




P= .000 


P= 


.555 


P = 


.000 


P= 


.000 


RDABL593 


.8529 




.1263 




.2947 




.4950 




( 632) 


( 


590) 


( 


592) 


( 


551) 




P= .000 


P= 


.002 


P = 


.000 


P= 


.000 


RDCHA593 


.7742 




.0388 




.1735 




.3309 




( 632) 


( 


586) 


( 


589) 


( 


551) 




P= .000 


P= 


.348 


P = 


.000 


P= 


.000 


RDABL394 


.3609 




.7496 




.2599 




.5477 




( 597) 


( 


619) 


( 


590) 


( 


551) 




P= .000 


P= 


.000 


P = 


.000 


P= 


.000 


RDCHA394 


.0699 




.6915 




.0825 




.3259 




( 589) 


( 


619) 


( 


583) 


( 


551) 




P= .090 


P= 


.000 


P = 


.047 


P= 


.000 


RDABL594 


.2661 




.7981 




.2910 




.5833 




( 604) 


( 


619) 


( 


598) 


( 


551) 




P= .000 


P= 


.000 


P = 


.000 


P= 


.000 


RDCHA594 


-.1325 




.7464 




.1447 




.4097 




( 597) 


( 


619) 


( 


593) 


( 


551) 




P= .001 


P= 


.000 


P = 


.000 


P= 


.000 


RDABL395 


.2750 




.4293 




.7796 




.8205 




( 603) 


( 


591) 


( 


625) 


( 


551) 




P= .000 


P= 


.000 


P = 


.000 


P= 


.000 


RDCHA395 


.1197 




.0499 




.7625 




.6287 




( 607) 


( 


596) 


( 


625) 


( 


551) 




P= .003 


P= 


.224 


P = 


.000 


P= 


.000 


RDABL595 


.2875 




.2366 




.8073 




.8374 




( 598) 


( 


591) 


( 


625) 


( 


551) 




P= .000 


P= 


.000 


P = 


.000 


P= 


.000 


RDCHA595 


.2067 




.0528 




.7677 




.6442 




( 598) 


( 


590) 


( 


625) 


( 


551) 




P= .000 


P= 


.200 


P = 


.000 


P= 


.000 


SELECT93 


1.0000 




.1389 




.3327 




.5342 




( 632) 


( 


583) 


( 


586) 


( 


551) 




P= . 


P= 


.001 


P = 


.000 


P= 


.000 


SELECT94 


.1389 


1 


.0000 




.2499 




.5966 




( 583) 


( 


619) 


( 


577) 


( 


551) 




P= .001 


P= 


• 


P = 


.000 


P= 


.000 


SELECT95 


.3327 




.2499 


1 


.0000 




.9064 




( 586) 


( 


577) 


( 


625) 


( 


551) 




P= .000 


P= 


.000 


P = 


. 


P= 


.000 


SELECT 


.5342 




.5966 




.9064 


1 


.0000 




( 551) 


( 


551) 


( 


551) 


( 


551) 




P= .000 


P= 


.000 


P = 


.000 


P= 


. 


(Coefficient 


; / (Cases) 


/ 


2-tailed 


Significance) 





is printed if a coefficient cannot be computed 
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3. We then grouped schools into income levels. 

PROCESS: Using the variable Percent Eligible for Free or Reduced Price 

Meals, the schools were grouped into quint ilds, low to high. The lowest 20% 
were categorized as high-income, the third 20% as moderate income, and the 
highest 20% as low income. Thus, the quintiles were treated as: 



First 20 Percent: 
Second 20 Percent: 
Third 20 Percent: 
Fourth 20 Percent: 
Fifth 20 Percent: 



High Income 
Ignored 

Moderate Income 

Ignored 

Low Income 



4. The schools were next grouped into urbanicity levels. 

PROCESS: Using National Center for Education Statistics locale codes, 

we grouped schools into urban, suburban, or rural. The table below shows how 
the grouping was done and the frequencies of schools in each of the groups. 



Code 


1 


(Large City) : 




Urban 


N=55 


Code 


2 


(Mid-Size City): 




Ignore 


N=17 


Code 


3 


(Urban Fringe of 


a Large City) : 


Suburban 


N=336 


Code 


4 


(Urban Fringe of 


a Mid-Size City) : 


Ignore 


N=13 


Code 


5 


(Large Town): 




Ignore 


N=4 


Code 


6 


(Small Town): 




Rural 


N=41 


Code 


7 


(Rural ) : 




Rural 


N=85 




9 



5. We then identified five groups from which to select one triad of schools, 
each: 



6. We then selected schools. 

PROCESS: In each group, we ordered the schools on the basis of the 

selection index. The two highest-scoring schools and the lowest-scoring 
school were invited to participate. In case a school did not agree, the next- 
higher scoring school or the next-lower scoring school were used as back-ups 
until the triad for that group was selected. These selections were discussed 
with Maryland State Department of Education personnel to find out if any 
concerns existed that should argue against selection any of these particular 
schools for study. Although several of the schools had already been singled 
out in one way or another, it was decided not to allow that to influence the 
choice of field study sites. Thus, no school was dropped from the study for 
reasons external to the selection process just described. 
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Appendix A 



School-Level Successfulness Indices Across 
Maryland School Performance Assessment Program Content Area Scores 

We applied analyses similar to those done on reading to the other five 
MSPAP content area scores. Besides these analyses, we also looked at 
regressions using the prior year mean as an additional predictor (as we did 
with reading) as well as, for fifth grade, using the two-year-ago mean for 
third grade (since there should be substantial overlap in students). Neither 
of these indices showed much stability over years and so we abandoned further 
modeling using prior achievement as a predictor. These analyses are available 
upon request. 

Purpose: To compare residual variation on school-level MSPAP content 

area scores for the six content area variables and composites across years. 

Sample: All Maryland elementary schools in 1993, 1994, and 1995. 

Method: Using weighted regression, weighting by the inverse variance of 

error of the mean, grade 3 and 5 school outcome means for each content area 
test score at each year were predicted from: 

elementary enrollment for that year 

elementary attendance rate for that year 

percent entrants for that year 

percent withdrawals for that year 

percent absent less than five days for that year 

percent absent more than twenty days for that year 

percent special education for that year 

percent free or reduced price means for that year 

percent Indian (American or Alaskan Native) for that year 

percent Asian or Pacific Islander for that year 

percent African American for that year 

percent Hispanic for that year 

percent accounted for on that year's MSPAP test for that content area at 
that grade level 

This resulted in 36 regressions (6 content areas by 2 grades by 3 
years). The studentized residuals from each equation were retained for 
further analyses. Composite variables were created as sums of residuals 
across grades and content areas. Numbers of school ranged from 605 to 711 
across the regressions. 

Results: Tables 1-6 show the intercorrelations among the residuals for 

each content area separately. The first four characters in each variable name 
identify the content area, the next (fifth) character identifies the grade 
level, and the next two (sixth and seventh) identify the year. 

Stability of residuals over grades is a methodological precondition to 
interpreting them as indices of higher or lower successfulness for schools. 

If they are specific to grades within schools but not stable across grades, 
then teacher cohorts would dominate school comparisons as Mandeville (1988) 
has found. Accordingly, Tables 1-6 were evaluated for grade stability. 

11 
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For Language, the correlations between grades 3 and 5 were: 

1993: .36 1994: .21 1995: .42 

For Reading, the correlations between grades 3 and 5 were: 

1993: .43 1994: .39 1995: .33 

For Writing, the correlations between grades 3 and 5 were: 

1993: .37 1994: .46 1995: .44 

For Social Studies, the correlations between grades 3 and 5 were: 

1993: .50 1994: .40 1995: .37 

For Science, the correlations between grades 3 and 5 were: 

1993: .45 1994: .48 1995: .44 

For Math, the correlations between grades 3 and 5 were: 

1993: .38 1994: .35 1995: .55 

These results suggest moderate stability between the third and fifth 
grade residuals for each content area. It seems appropriate to aggregate 
residuals across schools to assess successfulness. 

The content area residuals were then averaged for each school to form a 
content area composite. Table 7 shows the intercorrelations among these 6 
composites for the 3 years (18 variables). The correlations among these 
composites are substantial and in all cases statistically greater than zero. 

These composites were then averaged across content areas for each school 
separately for Verbal (Language, Reading, Writing, Social Studies) and 
Quantitative (Science, Math) subject matter areas. An average of these two 
scores was also created (called SEL) . Table 8 shows the intercorrelations 
among these composites. 

The correlations between Verbal and Quantitative areas were: 

1993: .85 1994: .86 1995: .88 

These correlations suggest that it is reasonable to combine the verbal and 
quantitative composites and form a composite for each school by averaging 
across the six content areas across the two grade levels. The SEL index is 
that composite. 

Intercorrelations among the SEL index across the three years ranged from 
.57 to .68. This suggests that the index is relatively stable, tending to 
rank schools similarly on a year-to-year basis. 

Conclusions 

The residuals appear reasonably stable across years. Not surprisingly, 
there do seem to be cohort effects, such that the correlations across content 
areas for the same year tend to be greater than for different years. 

The content areas do not seem to separate into groups according to 
patterns of intercorrelations, which led us to combine all six into a 

12 
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composite index (called SEL). The stability of the composite is probably due 
to one or both of two factors: (1) characteristics of school populations 

unmeasured (or not adequately measured) by the set of predictor variables , and 
(2) consistency of school effects. Examples of the former might be 
community-based programs, land uses, access to libraries, degree of crime, or 
transportation patterns. Examples of the latter might be educational 
backgrounds of the teachers, style of the principal, expenditure of resources, 
familiarity with MSPAP, or general school attitudes. 
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Table 1 . 
Scores 


Inter correlations 


of School 


Residuals on MSPAP Language Content 


Variable 


Cases 


Mean 


Std Dev 


LANG3930 


665 


.2137 


1.3049 


LANG5930 


608 


.0416 


1.2320 


LANG3940 


676 


.1060 


1.1942 


LANG5940 


594 


.0073 


1.5952 


LANG3950 


697 


.0578 


1.3572 


LANG5950 


605 


.0426 


1.2901 







- - Correlation Coefficients - - 








LANG3930 


LANG5930 


LANG3940 


LANG5940 


LANG3950 


LANG5950 


LANG3930 


1.0000 


.3635** 


.4283** 


.3555** 


.4323** 


.3839** 


LANG5930 


.3635** 


1.0000 


.2531** 


.4465** 


.2700** 


.2965** 


LANG3940 


.4283** 


.2531** 


1.0000 


.2137** 


.4285** 


.2066** 


LANG5940 


.3555** 


.4465** 


.2137** 


1.0000 


.1775** 


.4807** 


LANG3950 


.4323** 


.2700** 


.4285** 


.1775** 1 


.0000 


.4194** 


LANG5950 


.3839** 


.2965** 


.2066** 


.4807** 


.4194** 


1.0000 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


(2-tailed) 






" . " is printed if a 


coefficient 


cannot be 


computed 






Fourth character in variable name 


is grade; 


next two are 


year. 





Table 2. 


Intercorrelations 


of School 


Residuals on MSPAP Reading Content 


Scores 








Variable 


Cases 


Mean 


Std Dev 


READ3930 


688 


-.1434 


1.4306 


READ5930 


645 


-.1678 


1.4950 


READ3940 


660 


-.1343 


1.2396 


READ5940 


651 


-.1040 


1.6332 


READ3950 


683 


-.0752 


1.2033 


READ5950 


651 


-.1259 


1.4962 







- - Correlation Coefficients - - 






READ3930 


READ 5930 


READ 3 9 40 


READ5940 READ3950 


READ 59 50 


READ3930 


1.0000 


.4344** 


. 4456** 


.2413** .3156** 


.2814** 


READ5930 


.4344** 


1.0000 


.2427** 


.2916** .1829** 


.3203** 


READ3940 


. 4456** 


.2427** 


1.0000 


.3923** .4218** 


.1277** 


READ5940 


.2413** 


.2916** 


.3923** 


1.0000 .1291** 


.3283** 


READ3950 


.3156** 


.1829** 


.4218** 


.1291** 1.0000 


.3347** 


READ5950 


.2814** 


.3203** 


.1277** 


.3283** .3347** 


1.0000 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


(2-tailed) 




" . " is printed if a 


coefficient 


cannot be 


computed 





Fourth character in variable name is grade; next two are year. 
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Table 3. 
Scores 


Intercorrelations 


of School 


Residuals on MSPAP Writing Content 


Variable 


Cases 


Mean 


Std Dev 


WRIT3930 


678 


.1188 


1.1089 


WRIT5930 


626 


.0773 


1.3126 


WRIT3940 


695 


.1082 


1.1639 


WRIT5940 


671 


.0675 


1.2043 


WRIT3950 


689 


.0643 


1.2714 


WRIT5950 


626 


.1465 


1.2635 







- - Correlation Coefficients - - 








WRIT3930 


WRIT5930 


WRIT3940 


WRIT5940 


WRIT3950 


WRIT5950 


WRIT3930 


1.0000 


.3868** 


.4768** 


.4004** 


.3687** 


. 3334** 


WRIT5930 


.3868** 


1.0000 


.2631** 


.3554** 


.2797** 


.2175** 


WRIT3940 


.4768** 


.2631** 


1.0000 


.4618** 


.3632** 


.2964** 


WRIT5940 


.4004** 


.3554** 


.4618** 


1.0000 


.3465** 


.4881** 


WRIT3950 


.3687** 


.2797** 


.3632** 


.3465** 1 


.0000 


.4380** 


WRIT5950 


.3334** 


.2175** 


.2964** 


.4881** 


.4380** 


1.0000 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


( 2-tailed) 






" . " is printed if a 


coefficient 


cannot be 


computed 






Fourth character in variable name 


is grade; 


next two are 


year. 





Table 4* Intercorrelations of School Residuals on MSPAP Social Studies 
Content Scores 



Variable 


Cases 


Mean 


Std Dev 


SOCS3930 


711 


-.1202 


1.2855 


SOCS5930 


641 


-.0906 


1.1859 


SOCS3940 


707 


-.2122 


1.3195 


SOCS5940 


663 


-.1607 


1.3888 


SOCS3950 


710 


-.1052 


1.8408 


SOCS5950 


637 


-.1906 


2.0603 







- - Correlation Coefficients - - 






SOCS3930 


SOCS5930 


SOCS3940 


SOCS5940 SOCS3950 


SOCS5950 


SOCS3930 


1.0000 


.4958** 


.4991** 


.3012** .4068** 


.3002** 


SOCS5930 


.4958** 


1.0000 


.3575** 


.3401** .2633** 


.2190** 


SOCS3940 


.4991** 


.3575** 


1.0000 


.3951** .3587** 


.2447** 


SOCS5940 


.3012** 


.3401** 


.3951** 


1.0000 .2178** 


.4079** 


SOCS3950 


.4068** 


.2633** 


.3587** 


.2178** 1.0000 


. 3664** 


SOCS5950 


.3002** 


.2190** 


.2447** 


.4079** .3664** 


1.0000 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


(2-tailed) 





" . " is printed if a coefficient cannot be computed 

Fourth character in variable name is grade; next two are year. 
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Table 5* Intercorrelations of School Residuals on MSPAP Science Content 
Scores 



Variable 


Cases 


Mean 


Std Dev 


SCIN3930 


700 


-.1796 


1.7239 


SCIN5930 


648 


-.0999 


1.3121 


SCIN3940 


698 


-.1316 


1.2232 


SCIN5940 


655 


-.0592 


1.3880 


SCIN3950 


700 


-.1254 


1.6803 


SCIN5950 


626 


-.0869 


1.5213 







- - Correlation Coefficients - - 








SCIN3930 


SCIN5930 


SCIN3940 


SCIN5940 


SCIN3950 


SCIN5950 


SCIN3930 


1.0000 


.4514** 


.4130** 


.2286** 


.2142** 


.3565** 


SCIN5930 


.4514** 


1.0000 


.3176** 


.4271** 


.2242** 


.2612** 


SCIN3940 


.4130** 


.3176** 


1.0000 


.4789** 


.3860** 


.3651** 


SCIN5940 


.2286** 


.4271** 


.4789** 


1.0000 


.2989** 


.4641** 


SCIN3950 


.2142** 


.2242** 


.3860** 


.2989** 1 


.0000 


.4368** 


SCIN5950 


.3565** 


.2612** 


.3651** 


.4641** 


.4368** 


1.0000 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


(2-tailed) 






" . " is printed if a 


coefficient 


cannot be 


computed 






Fourth character in variable name 


is grade; 


next two are 


year * 





Table 6. 


Inter cor relations 


of School 


Residuals on MSPAP Math Content Scores 


Variable 


Cases 


Mean 


Std Dev 


MATH3930 


683 


-.3220 


2.6661 


MATH5930 


678 


-.0483 


1.1200 


MATH3940 


679 


-.1877 


1.7891 


MATH5940 


668 


-.1144 


1.4897 


MATH3950 


677 


-.1697 


1.4019 


MATH5950 


630 


-.0428 


1.3094 







- - Correlation Coefficients - - 








MATH3930 


MATH5930 


MATH3940 


MATH5940 


MATH3950 


MATH5950 


MATH3930 


1.0000 


.3832** 


.3132** 


.2131** 


.2580** 


.1842** 


MATH5930 


.3832** 


1.0000 


.3203** 


.4313** 


.3179** 


.3924** 


MATH3940 


.3132** 


.3203** 


1.0000 


.3463** 


.3959** 


.2712** 


MATH5940 


.2131** 


.4313** 


.3463** 


1.0000 


.2675** 


.4535** 


MATH3950 


.2580** 


.3179** 


.3959** 


.2675** 1 


.0000 


.5536** 


MATH5950 


.1842** 


.3924** 


.2712** 


.4535** 


.5536** 


1.0000 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


(2-tailed) 







" . " is printed if a coefficient cannot be computed 

Fourth character in variable name is grade; next two are year. 
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Table 7. Intercorrelations of Sums of School Residuals Across Third and Fifth 
Grades on MSPAP Content Scores 



Variable 


Cases 


Mean 


Std Dev 


LANG93 


605 


.1152 


.9877 


LANG94 


591 


.0493 


1.0076 


LANG95 


604 


.0497 


.9551 


RE AD 9 3 


641 


-.1549 


1.0724 


READ94 


632 


-.1242 


.9948 


READ95 


644 


-.1034 


1.0457 


WRIT93 


624 


.0898 


.9704 


WRIT94 


666 


.0832 


1.0008 


WRIT95 


624 


.1164 


.9674 


SOCS93 


640 


-.1278 


.9646 


SOCS94 


660 


-.1712 


1.0504 


SOCS95 


635 


-.1648 


1.3232 


SCIN93 


645 


-.1569 


1.0978 


SCIN94 


651 


-.0831 


1.0831 


SCIN95 


624 


-.0857 


1.1208 


MATH93 


657 


-.1987 


1.6211 


MATH 9 4 


652 


-.1163 


1.2207 


MATH 9 5 


624 


-.1043 


1.0712 



Last two characters in variable name are year. 
Correlations are on the next page. 
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LANG93 


LANG94 


LANG95 


READ93 


READ94 


READ95 


LANG93 


1 .0000 


.5962** 


.5136** 


.6180** 


.4606** 


.3513** 


LANG94 


.5962** 


1 .0000 


.5661** 


.4904** 


.5847** 


.4829** 


LANG95 


.5136** 


.5661** 


1 .0000 


.3813** 


.4607** 


.7624** 


READ93 


.6180** 


.4904** 


.3813** 


1 .0000 


.4964** 


.4037** 


READ94 


.4606** 


.5847** 


.4607** 


.4964** 


1 .0000 


.5041** 


READ95 


.3513** 


.4829** 


.7624** 


.4037** 


.5041** 


1 .0000 


WRIT93 


.8210** 


.5685** 


.5033** 


.6866** 


.4594** 


.3699** 


WRIT94 


.4818** 


.6715** 


.4499** 


.3777** 


.7299** 


.4485** 


WRIT95 


.4929** 


.5782** 


.8342** 


.4168** 


.5032** 


.7501** 


S0CS93 


.6844** 


.5234** 


.4649** 


.8806** 


.5403** 


.4124** 


S0CS94 


.4329** 


.6396** 


.4921** 


.4554** 


.8450** 


.5140** 


S0CS95 


.3397** 


.4591** 


.6655** 


.4128** 


.4130** 


.8121** 


SCIN93 


.6260** 


.5047** 


.4201** 


.8175** 


.5322** 


.3788** 


SCIN94 


.4169** 


.5956** 


.4595** 


.4288** 


.8150** 


.5125** 


SCIN95 


.3427** 


.5024** 


.7208** 


.4385** 


.4723** 


.8585** 


MATH93 


.5666** 


.4518** 


.2736** 


.6096** 


.5011** 


.2600** 


MATH94 


.3531** 


.5862** 


.3936** 


.3698** 


.7677** 


.4386** 


MATH95 


.3334** 


.4459** 


.6575** 


.4454** 


.4722** 


.7952** 




WRIT93 


WRIT94 


WRIT95 


S0CS93 


S0CS94 


S0CS95 


LANG93 


.8210** 


.4818** 


.4929** 


.6844** 


.4329** 


.3397** 


LANG94 


.5685** 


.6715** 


.5782** 


.5234** 


.6396** 


.4591** 


LANG95 


.5033** 


.4499** 


.8342** 


.4649** 


.4921** 


.6655** 


READ93 


.6866** 


.3777** 


.4168** 


.8806** 


.4554** 


.4128** 


READ94 


.4594** 


.7299** 


.5032** 


.5403** 


.8450** 


.4130** 


READ95 


.3699** 


.4485** 


.7501** 


.4124** 


.5140** 


.8121** 


WRIT93 


1 .0000 


.5298** 


.4415** 


.7276** 


.4192** 


.3474** 


WRIT94 


.5298** 


1 .0000 


.5360** 


.4853** 


.7266** 


.4178** 


WRIT95 


.4415** 


.5360** 


1 .0000 


.4564** 


.4962** 


.6772** 


S0CS93 


.7276** 


.4853** 


.4564** 


1 .0000 


.5074** 


.4070** 


S0CS94 


.4192** 


.7266** 


.4962** 


.5074** 


1 .0000 


.5135** 


S0CS95 


.3474** 


.4178** 


.6772** 


.4070** 


.5135** 


1 .0000 


SCIN93 


.6962** 


.4478** 


.4494** 


.8642** 


.4901** 


.4180** 


SCIN94 


.4007** 


.7007** 


.5251** 


.5095** 


.8713** 


.4821** 


SCIN95 


.3805** 


.4366** 


.7637** 


.4659** 


.5419** 


.9118** 


MATH 93 


.5908** 


.4138** 


.2944** 


.6073** 


.4997** 


.2962** 


MATH 94 


.3694** 


.5759** 


.3793** 


.4400** 


.7318** 


.4486** 


MATH95 


.3417** 


.4315** 


.6882** 


.4574** 


.5769** 


.9038** 




SCIN93 


SCIN94 


SCIN95 


MATH93 


MATH94 


MATH95 


LANG93 


.6260** 


.4169** 


.3427** 


.5666** 


.3531** 


.3334** 


LANG94 


.5047** 


.5956** 


.5024** 


.4518** 


.5862** 


.4459** 


LANG95 


.4201** 


.4595** 


.7208** 


.2736** 


.3936** 


.6575** 


READ93 


.8175** 


.4288** 


.4385** 


.6096** 


.3698** 


.4454** 


READ94 


.5322** 


.8150** 


.4723** 


.5011** 


.7677** 


.4722** 


READ95 


.3788** 


.5125** 


.8585** 


.2600** 


.4386** 


.7952** 


WRIT93 


.6962** 


.4007** 


.3805** 


.5908** 


.3694** 


.3417** 


WRIT94 


.4478** 


.7007** 


.4366** 


.4138** 


.5759** 


.4315** 


WRIT95 


.4494** 


.5251** 


.7637** 


.2944** 


.3793** 


.6882** 


S0CS93 


.8642** 


.5095** 


.4659** 


.6073** 


.4400** 


.4574** 


S0CS94 


.4901** 


.8713** 


.5419** 


.4997** 


.7318** 


.5769** 


S0CS95 


.4180** 


.4821** 


.9118** 


.2962** 


.4486** 


.9038** 


SCIN93 


1 .0000 


.5339** 


.4690** 


.7397** 


.4582** 


.4158** 


SCIN94 


.5339** 


1 .0000 


.5492** 


.5088** 


.8050** 


.5772** 


SCIN95 


.4690** 


.5492** 


1 .0000 


.3146** 


.4974** 


.8862** 


MATH93 


.7397** 


.5088** 


.3146** 


1 .0000 


.4630** 


.3483** 


MATH94 


.4582** 


.8050** 


.4974** 


.4630** 


1 .0000 


.5301** 


MATH95 


.4158** 


.5772** 


.8862** 


.3483** 


.5301** 


1 .0000 



o 

ERIC 
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Table 8. Intercorrelations of Sums of School Residuals Across 

Verbal MSPAP Content Scores (Language + Reading + Writing + Social Studies), 



Quantitative 


MSPAP Content 


Scores (Science 


+ Math) , and 


Combined MSPAP Content Scores (Verbal + Quantitative) 


Variable 


Cases 


Mean 


Std Dev 


VERB93 


605 


-.0159 


.8064 


VERB94 


591 


-.0352 


.7544 


VERB95 


600 


-.0035 


.8683 


QUAN93 


644 


-.1694 


1.2163 


QUAN94 


646 


-.0968 


1.0750 


QUAN95 


622 


-.0913 


1.0542 


SEL93 


605 


-.0497 


.8000 


SEL94 


591 


-.0685 


.7630 


SEL95 


598 


-.0174 


.8404 







- - Correlation Coefficients - - 








VERB93 


VERB94 


VERB95 


QUAN93 


QUAN94 


QUAN95 


VERB93 


1.0000 


.6652** 


.5637** 


.8526** 


.5071** 


.4773** 


VERB94 


.6652** 


1.0000 


.6231** 


.6025** 


.8628** 


. 5629** 


VERB95 


.5637** 


.6231** 


1.0000 


.4177** 


.5409** 


.8771** 


QUAN93 


.8526** 


.6025** 


.4177** 


1.0000 


.5504** 


.4079** 


QUAN94 


.5071** 


.8628** 


.5409** 


.5504** 1 


.0000 


.5885** 


QUAN95 


.4773** 


.5629** 


.8771** 


.4079** 


.5885** 


1.0000 


SEL93 


.9818** 


.6760** 


.5598** 


.9363** 


. 5484** 


.4958** 


SEL94 


.6576** 


.9820** 


.6205** 


.6365** 


.9428** 


.6019** 


SEL95 


• 5664 * * 


.6264** 


.9847** 


.4475** 


.5795** 


.9473** 




SEL93 


SEL94 


SEL95 








VERB93 


.9818** 


.6576** 


• 5 6 64 * * 








VERB94 


.6760** 


.9820** 


.6264** 








VERB95 


.5598** 


.6205** 


.9847** 








QUAN93 


.9363** 


.6365** 


.4475** 








QUAN94 


.5484** 


.9428** 


.5795** 








QUAN95 


.4958** 


.6019** 


.9473** 








SEL93 


1.0000 


.6827** 


.5735** 








SEL94 


.6827** 


1.0000 


.6385** 








SEL95 


.5735** 


.6385** 


1.0000 








* - Signif. 


LE .05 


** - Signif 


. LE .01 


( 2-tailed) 







is printed if a coefficient cannot be computed 



Last two characters in variable name are year. 
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Appendix B 



Comparison of the Combined MSPAP Content Score Residuals 
with the MSDE School Performance Index and Its Residuals 



While the Maryland School Performance Assessment Program is usually 
called "high stakes" for schools, decisions are actually reached by the 
Maryland State Department of Education (MSDE) on the basis of the School 
Performance Index* The MSDE School Performance Index (SPI) for an elementary 
school is the average of 13 ratios, each of which is the observed percentage 
of a variable at a school divided by the targeted percentage for school 
performance to be satisfactory for that variable* The thirteen variables and 
targeted percents are: 

Attendance Rate 94 



Grade 


3 


MSPAP 


Percent 


Satisfactory 


in 


Reading 


70 


Grade 


3 


MSPAP 


Percent 


Satisfactory 


in 


Writing 


70 


Grade 


3 


MSPAP 


Percent 


Satisfactory 


in 


Language Usage 


70 


Grade 


3 


MSPAP 


Percent 


Satisfactory 


in 


Mathematics 


70 


Grade 


3 


MSPAP 


Percent 


Satisfactory 


in 


Science 


70 


Grade 


3 


MSPAP 


Percent 


Satisfactory 


in 


Social Studies 


70 


Grade 


5 


MSPAP 


Percent 


Satisfactory 


in 


Reading 


70 


Grade 


5 


MSPAP 


Percent 


Satisfactory 


in 


Writing 


70 


Grade 


5 


MSPAP 


Percent 


Satisfactory 


in 


Language Usage 


70 


Grade 


5 


MSPAP 


Percent 


Satisfactory 


in 


Mathematics 


70 


Grade 


5 


MSPAP 


Percent 


Satisfactory 


in 


Science 


70 


Grade 


5 


MSPAP 


Percent 


Satisfactory 


in 


Social Studies 


70 



In order to study the SPI in relation to the combined MSPAP content area 
score residuals, the SPI was calculated for each school for each year, 1993, 
1994, and 1995* Residual variation in the SPI was studied by predicting it 
using the same predictors employed in the content area predictions earlier 
(percents accounted for on all the content areas for both grades were treated 
as predictors since the SPI includes all content areas; not surprisingly, 
several were not possible to enter in each regression since they were linearly 
related perfectly with others that were entered)* 

Because attendance rate was a predictor in the earlier content area 
predictions but is also a factor in the SPI, predictions were made with and 
without attendance rate in the predictor set* This resulted in six variables, 
RESSPI93 , RESSPI94 , and RESSPI95 are the studentized residuals from the 
regressions calculated without attendance rate and RESSCI93, RESSCI94, and 
RESSCI95 are the studentized residuals calculated with attendance rate in the 
predictor set* 

The multiple R squares for these six regressions were: 



Attendance Rate 



Year 


With 


Without 


1993 


.721 


*719 


1994 


.732 


*732 


1995 


.717 


*716 



20 



0 
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The small differences between the R squares with and without attendance 
rate as a predictor suggests that results from the MSPAP test scores are the 
dominant determiners of variation in the SPI. 

The table below contains means, standard deviations, and correlations of 
the SPI and the studentized residuals of both it and the overall selection 
index developed on the basis of combining residuals of MSPAP means. The raw 
SPI correlates about .9 across years. That these correlations are stronger 
than the intercorrelations of the other variables, suggests that the SPI is 
sensitive to variables that were used as controls in the regressions. The 
strong within-year R squares between the controls and the SPI are 
corroborating evidence. 

The differences between the SPI-based residuals and the combined 
residualized MSPAP content scores are of interest. There are two fundamental 
methodological differences between these sets of residuals. First, the SPI is 
calculated from proportions above fixed cut-off points on the MSPAP content 
area scales, while MSPAP content area means were used to arrive at 
residualized content scores. Since the cut-off points are in relatively dense 
regions of the scales (about .6 standard deviations above the mean on the 1991 
norming sample), the means and the proportions may be more interchangeable 
currently than in the future if average performance drifts appreciably in 
either direction. 

The second methodological difference has to do with the weighting used 
in the regressions when predicting school means on the content area scores. 
Each school received a weight that was the inverse of the estimated sampling 
variance of its mean. However, no weight was used in regressions predicting 
the SPI since its standard error is a far more complicated statistic. 

The correlations suggest that residuals based on the SPI are relatively 
interchangeable with combined residuals based on content area means in any one 
year, all being about .9 whether or not attendance rate is in the predictor 
set. The most stable across years were residuals based on SPI predicted 
without attendance rate as a predictor (average intercorrelation of RESSPI = 
.68), the next most stable were residuals based on SPI predicted with 
attendance rate as a predictor (average intercorrelations of RESSCI = *67), 
and the least stable were average content area residual (average 
intercorrelation of SEL = .63), but these are not very different. 
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* r '$ 



Means , Standard Deviations, and Intercorrelations of the School Performance 
Index ( SPI ) , Residualized SPI Without Attendance as a Predictor (RESSPI), 
Residualized SPI With Attendance as a Predictor (RESSCI), and Combined 
Residualized MSPAP Content Scores Across Years* 



Variable 


Cases 


Mean 


Std Dev 


SPI93 


734 


• 4800 


.2092 


SPI94 


734 


• 5222 


.2184 


SPI95 


734 


• 5821 


.2308 


RESSPI93 


731 


.0009 


1.0023 


RESSPI94 


730 


• 0011 


1.0020 


RESSPI95 


728 


• 0007 


1.0014 


RESSCI93 


731 


.0009 


1.0020 


RESSCI94 


730 


• 0012 


1.0020 


RESSCI95 


728 


.0007 


1.0016 


SEL93 


605 


-.0497 


. 8000 


SEL94 


591 


-.0685 


.7630 


SEL95 


598 


-.0174 


.8404 







- - Correlation Coefficients 


— 






SPI93 


SPI94 


SPI95 


RESSPI93 


RESSPI94 


RESSPI95 


SPI93 


1.0000 


.9193** 


.8976** 


.5275** 


.3691** 


.3435** 


SPI94 


.9193** 


1.0000 


.9100** 


.3927** 


.5158** 


.3539** 


SPI95 


.8976** 


.9100** 


1.0000 


.3793** 


.3802** 


.5316** 


RESSPI93 


.5275** 


.3927** 


.3793** 


1.0000 


.7020** 


. 6364** 


RESSPI94 


.3691** 


.5158** 


.3802** 


.7020** 


1.0000 


. 6834** 


RESSPI95 


.3435** 


.3539** 


.5316** 


.6364** 


. 6834** 


1.0000 


RESSCI93 


.5262** 


.3930** 


.3790** 


.9975** 


.7028** 


.6371** 


RESSCI94 


.3690** 


.5156** / 


.3798** 


.7016** 


.9998** 


.6831** 


RESSCI95 


.3421** 


.3526** 


.5308** 


.6326** 


. 6800** 


.9991** 


SEL930 


.4699** 


.3588** 


.3341** 


.8886** 


.6619** 


.5936** 


SEL940 


.3461** 


.4742** 


.3551** 


.6271** 


.9028** 


.6321** 


SEL950 


.2471** 


.2668** 


.4356** 


.5426** 


. 6086** 


.9104** 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


(2-tailed) 




" . " is printed if a 


coefficient 


cannot be 


computed 








RESSCI93 


RESSCI94 


RESSCI95 


SEL93 


SEL94 


SEL95 


SPI93 


.5262** 


.3690** 


.3421** 


.4699** 


.3461** 


.2471** 


SPI94 


.3930** 


.5156** 


.3526** 


.3588** 


.4742** 


.2668** 


SPI95 


.3790** 


.3798** 


.5308** 


.3341** 


.3551** 


.4356** 


RESSPI93 


.9975** 


.7016** 


.6326** 


.8886** 


.6271** 


.5426** 


RESSPI94 


.7028** 


.9998** 


.6800** 


.6619** 


.9028** 


.6086** 


RESSPI95 


.6371** 


.6831** 


.9991** 


.5936** 


.6321** 


.9104** 


RESSCI93 


1.0000 


.7021** 


.6339** 


.8904** 


.6283** 


.5423** 


RESSCI94 


.7021** 


1.0000 


.6795** 


.6609** 


.9026** 


. 6083** 


RESSCI95 


.6339** 


.6795** 


1.0000 


.5913** 


.6290** 


.9104** 


SEL93 


.8904** 


.6609** 


.5913** 


1.0000 


.6827** 


.5735** 


SEL94 


.6283** 


.9026** 


.6290** 


.6827** 


1.0000 


.6385** 


SEL95 


.5423** 


. 6083** 


.9104** 


.5735** 


.6385** 


1.0000 


* - Signif. 


LE .05 


** - Signif 


. LE .01 


(2-tailed) 
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